Image processing system and image processing program

ABSTRACT

In the present invention, the following are provided: a 3D information generating unit ( 3 ) for generating 3D information as the data of a group of a plurality of points formed by projecting the values of respective pixels of a moving object in accordance with depth information detected from an image pickup image; an overlooking image generating unit ( 4 ) for generating an overlooking image by synthesizing the 3D information of the moving object with a space image of an image pickup target region; and a display control unit ( 5 ) for displaying the overlooking image. The present invention is configured so that, even in a case where there are multiple image pickup target regions in a large-scale building in which the floor configuration is complicated, it is unnecessary to display multiple image pickup images using split screen display, and by displaying one overlooking image formed by synthesizing the 3D information of the moving objects formed from the groups of a plurality of points with each of a plurality of image pickup target regions which are included in the entire space of the building, the overall state of the building can be ascertained in one glance by confirming the overlooking image.

TECHNICAL FIELD

The present invention relates to an image processing system and an imageprocessing program and more particularly to an image processing systemand an image processing program including a function for detecting amoving object from an image photographed by a camera and displaying thedetected moving object as 3D information.

BACKGROUND ART

A conventional monitoring system displays a plurality of monitoringimages photographed by a plurality of monitoring cameras on a pluralityof image display devices or displays the monitoring images on a singleimage display device by division, thereby performing monitoring.Alternatively, a plurality of monitoring images is switched by timedivision and is displayed on a single image display device, therebyperforming monitoring. However, the conventional monitoring system isnot suitable for the case in which an inner part of a building having acomplicated structure is monitored.

In other words, a large-scaled building such as a factory or a buildinghas a complicated floor structure and a large number of rooms. In orderto monitor an inner part of the building having the complicatedstructure, a large number of cameras are provided so that the number ofmonitoring images to be displayed on an image display device is alsoincreased. For this reason, it is hard to grasp, at a glance, whichfloor and which room for a plurality of monitoring images displayed onthe image display device and it is difficult to understand a situationof the whole building.

In contrast, there is proposed a system for synthesizing monitoringimages photographed by monitoring cameras provided for each ofmonitoring regions of a floor with another prepared floor image, therebyenabling the monitoring images to be displayed on a single screen at atime in accordance with a layout of the floor (for example, see PatentDocument 1). According to the technique described in the Patent Document1, it is possible to simultaneously monitor monitoring regions such asan inner part of each room, a corridor or the like in a building througha single screen in accordance with a layout of a floor.

Referring to the technique described in the Patent Document 1, however,the monitoring images photographed by the monitoring cameras provided inrespective monitoring regions are simply allocated and displayed inpositions of the monitoring regions for a planar floor image shown in atop view. Accordingly, this is basically the same as division anddisplay of a plurality of monitoring images on a single image displaydevice to perform monitoring. Therefore, it is possible to grasp whichmonitoring region has the monitoring images photographed. However, thereis a problem in that it is necessary to individually confirm themonitoring images in order to understand a situation of a whole buildingand it is still hard for a user to use them.

Moreover, the technique described in the Patent Document 1 is notsuitable for monitoring a building having a large scale and acomplicated floor structure. In other words, the technique of the PatentDocument 1 serves to divide and display the monitoring images on asingle screen. For this reason, the number of images which can bedisplayed has a limit. As shown in FIG. 7 of the Patent Document 1,really, the monitoring images photographed in the monitoring regions inone floor can be simply displayed and cannot be used for whollymonitoring a large-scaled and complicated building such as a factory, abuilding or a department store. If a large number of monitoring imagesare to be forcibly displayed on a single screen, display sizes of theindividual monitoring images are reduced so that they are hard to see.

According to FIG. 9 in Japanese Laid-Open Patent Publication No.2008-118466 (Patent Document 2) cited as the prior art document inparagraph [0004] of the Patent Document 1, monitoring images in aplurality of rooms are synthetized to generate an overlooking imageevery floor (such an image as to look at each floor downward) andoverlooking images in the floors are further synthesized to generatefloor superposition images 1F to 3F. Consequently, a whole monitoringimage of a building having a plurality of floors or the like can bedisplayed on a single screen.

When a whole monitoring image of a large-scaled and complicated buildingis to be thus displayed on the single screen, however, display sizes ofthe monitoring images in the individual monitoring regions are reducedso that the monitoring images are very hard to see. Accordingly, it isalso difficult to say that the technique described in the PatentDocument 2 is suitable for monitoring a building having a large scaleand a complicated floor structure.

Patent Document 1: Japanese Laid-Open Patent Publication No. 2012-4630

Patent Document 2: Japanese Laid-Open Patent Publication No. 2008-118466

DISCLOSURE OF THE INVENTION

The present invention has been made to solve these problems and has anobject to enable an overlooking image of a building having a large scaleand a complicated floor structure to be offered so as to be easilyunderstood and to enable a user to readily grasp a situation of thewhole building.

In order to solve the problem, the present invention includes an imageinput unit for inputting photographed images respectively from aplurality of cameras provided to photograph a photographing targetregion at a plurality of angles, a depth information calculating unitfor calculating, every pixel, depth information representing distancesfrom the cameras of moving objects included in the input photographedimages, a 3D information generating unit for setting a plurality ofprojection planes in adaptation to a relative angular relationshipbetween the cameras and projecting values of each of pixels of themoving objects included in the photographed images in directions of theprojection planes corresponding to the respective photographed images inaccordance with the depth information, thereby generating 3D informationobtained by synthesizing, into one, the moving objects included in thephotographed images, an overlooking image generating unit forsynthesizing the 3D information of the moving object with a space imagerepresenting a space of the photographing target region, therebygenerating an overlooking image of the photographing target region, anda display control unit for displaying the generated overlooking image ona display.

According to the present invention having the structure described above,the 3D information of the moving objects are generated as data on groupsof a plurality of points formed by projecting the values of therespective pixels in the directions of the projection planes inaccordance with the depth information detected from the photographedimages, the 3D information are synthesized with the space image of thephotographing target region and the synthesized image is displayed as asingle whole overlooking image. For this reason, even if the number ofthe photographing target regions is increased in a building having alarge scale and a complicated floor structure, a large number ofphotographed images are neither displayed by screen division nordisplayed by switching through time division. In other words, a singleoverlooking image which has a plurality of photographing target regionsin a whole space of the building, and which is formed by synthesizingthe 3D information of the groups of a plurality of points in each of thephotographing target regions is displayed on a display.

Consequently, a user does not need to conventionally perform individualconfirmation of a plurality of photographed images displayed everyphotographing target region but can grasp a situation of the wholebuilding at a glance by confirming the overlooking image. Moreover, itis also possible to eliminate a disadvantage that the individualphotographed image displayed by the screen division is hard to see dueto a reduction in a display size thereof. According to the presentinvention, therefore, an overlooking image of a building having a largescale and a complicated floor structure can be offered so as to beeasily understood and the user can readily grasp the situation of thewhole building.

According to the present invention, moreover, the values of therespective pixels of the moving objects included in the respectivephotographed images obtained by photographing the photographing targetregion at a plurality of angles are projected from the photographedimages in the directions of the projection planes of the respectivephotographed images so that the 3D information are synthesized. Bymoving a position of a virtual viewpoint for seeing the overlookingimage including the 3D information of the moving objects to change anazimuth of the projection plane, therefore, it is possible to performoptional switching into an overlooking image seen at various angles,thereby displaying the overlooking image on a display.

In the related art, a single camera is provided in a singlephotographing target region. For this reason, it is possible to displayonly a photographed image in a direction in which the camera isprovided. Even if a plurality of cameras is provided in the singlephotographing target region, a plurality of photographed images obtainedby photographing the single photographing target region at a pluralityof angles can be simply displayed by screen division or time division.

In contrast, according to the present invention, the overlooking imageof the photographing target region including the 3D information of themoving objects processed into the data on the groups of a plurality ofpoints from the photographed images can be displayed on the display, andfurthermore, the viewpoint of the overlooking image can be optionallyswitched and displayed as described above. For example, consequently, anobject hidden behind something as seen at a certain angle can also bedisplayed with the angle of the overlooking image changed.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram showing an example of a structure of an imageprocessing system according to first and third embodiments.

FIG. 2 is a block diagram showing an example of a functional structureof the image processing system according to the first embodiment.

FIG. 3 is a view for explaining processing of a 3D informationgenerating unit according to the first to third embodiments.

FIG. 4 is a view showing an example of an overlooking image to bedisplayed on a display by a display control unit according to the firstto third embodiments.

FIG. 5 is a table showing an example of relevant information to bestored in a relevant information storing unit according to the first tothird embodiments.

FIG. 6 is a flowchart showing an example of an operation of an imageprocessing device to be one component of the image processing systemaccording to the first embodiment.

FIG. 7 is a diagram showing an example of a structure of the imageprocessing system according to the second embodiment.

FIG. 8 is a block diagram showing an example of a functional structureof the image processing system according to the second embodiment.

FIG. 9 is a view for explaining processing of a display target regionspecifying unit according to the second embodiment.

FIG. 10 is a block diagram showing an example of a functional structureof the image processing system according to the third embodiment.

FIG. 11 is a view showing an example of an alarm to be generated by analarm generating unit according to the third embodiment.

FIG. 12 is a view showing an example of a motion of a moving object tobe tracked by a moving object tracking unit according to the thirdembodiment.

BEST MODE FOR CARRYING OUT THE INVENTION First Embodiment

A first embodiment of the present invention will be described below withreference to the drawings. FIG. 1 is a diagram showing an example of astructure of an image processing system according to the firstembodiment. In the first embodiment, there is shown an example in whichthe image processing system according to the present invention isexecuted in a monitoring system.

As shown in FIG. 1, a monitoring system according to the firstembodiment includes a plurality of image pickup units 101, 102, . . . ,a plurality of image input devices 201, 202, . . . , an image processingdevice 300, and a display 400. The image pickup units 101, 102, . . .are connected to the image input devices 201, 202, . . . , respectively.Moreover, the image input devices 201, 202, . . . are connected to theimage processing device 300 through an in-house network 500 in abuilding.

The image pickup units 101, 102, . . . are provided every photographingtarget regions in the building. Herein, the photographing target regionsindicate desirable space regions to be monitoring targets, for example,inner parts of respective rooms, corridors, steps, elevators and thelike present in respective floors of the building. A single image pickupunit is allocated to a single photographing target region. In thefollowing, for convenience of explanation, description will be given onthe assumption that the two image pickup units 101 and 102 are provided.

A single image pickup unit includes a plurality of cameras. For example,the first image pickup unit 101 includes four cameras 101A to 101D. Thesecond image pickup unit 102 also includes a plurality of cameras, whichis not shown in FIG. 1. The number of the cameras provided in the secondimage pickup unit 102 does not need to be equal to that of the firstimage pickup unit 101.

The cameras 101A to 101D are provided to photograph the photographingtarget region at a plurality of angles. For example, in the case inwhich a certain room is set to be the photographing target region, wallsurfaces in four direction of the room are provided with the fourcameras 101A to 101D so as to enable a whole inner part of the room tobe photographed. In this case, it is preferable to set positions orpostures of the respective cameras 101A to 101D in such a manner thatrespective photographing ranges overlap with each other in at least apart thereof.

All of these cameras 101A to 101D are stereo cameras and simultaneouslyphotograph targets in the photographing target regions in two differentdirections. As will be described below, the image processing device 300analyzes the photographed image through a known technique, therebyenabling information in a depth direction of a target to be acquired.

A plurality of image input devices 201 and 202 serves to inputphotographed images from the image pickup units 101 and 102respectively, and is configured from a personal computer or the like,for example. Although the first image input device 201 is connected tothe first image pickup unit 101 and the second image input device 202 isconnected to the second image pickup unit 102 in the example of FIG. 1,the present invention is not restricted to such a connectionconfiguration. For example, the two image pickup units 101 and 102 maybe connected to a single image input device and a photographed image maybe input from the two image pickup units 101 and 102 to the single imageinput device.

The image processing device 300 inputs photographed images from theimage input devices 201 and 202 to perform image processing which willbe described below in detail. Then, the display 400 is caused to displayan image obtained as a result of the image processing. Herein, the imageprocessing device 300 is configured from a personal computer or a serverdevice, for example, and is provided in a monitoring room in a buildingor the like. Moreover, the display 400 is configured from a liquidcrystal display, for example.

In the system structure shown in FIG. 1, portions of the image inputdevices 201 and 202 and the image processing device 300 correspond tothe image processing system according to the present invention. FIG. 2is a block diagram showing an example of a functional structure of theimage processing system. As shown in FIG. 2, the image processing systemaccording to the first embodiment includes, as a functional structurethereof, an image input unit 1, a depth information calculating unit 2,a 3D information generating unit 3, an overlooking image generating unit4, a display control unit 5, a relevant information generating unit 6,an operation accepting unit 7, a space image storing unit 8 and arelevant information storing unit 9.

Herein, a function of the image input unit 1 is provided in the imageinput devices 201 and 202 shown in FIG. 1, respectively. Moreover,respective functions of the depth information calculating unit 2, the 3Dinformation generating unit 3, the overlooking image generating unit 4,the display control unit 5, the relevant information generating unit 6,the operation accepting unit 7, the space image storing unit 8 and therelevant information storing unit 9 are provided in the image processingdevice 300 shown in FIG. 1.

The respective functions of the depth information calculating unit 2,the 3D information generating unit 3, the overlooking image generatingunit 4, the display control unit 5, the relevant information generatingunit 6 and the operation accepting unit 7 provided in the imageprocessing device 300 can be configured from all of hardware, a DSP(Digital Signal Processor) and software. In the case in which thefunctions are configured from the software, for example, they actuallyinclude a CPU, an RAM, an ROM and the like in the image processingdevice 300 and are implemented by an operation of an image processingprogram stored in a recording medium such as the RAM, the ROM, a harddisk or a semiconductor memory.

The image input unit 1 inputs photographed images from a plurality ofcameras, respectively. In other words, the image input unit 1 providedin the first image input device 201 inputs photographed images from thecameras 101A to 101D, respectively. Moreover, the image input unit 1provided in the first image input device 201 inputs photographed imagesfrom a plurality of cameras (not shown in FIG. 1), respectively.

The depth information calculating unit 2 calculates, every pixel, depthinformation representing distances from cameras of moving objectsincluded in the photographed images input by the image input unit 1. Asdescribed above, the photographed image to be input by the image inputunit 1 is a parallax image photographed by a stereo camera. The depthinformation calculating unit 2 analyzes the parallax image by a knowntechnique, thereby calculating, every pixel, the depth informationrepresenting the distance from the camera of the moving object.

As a method of extracting the moving object from an inner part of thephotographed image, it is possible to apply various methods. Forexample, by detecting a difference between frame images to bephotographed sequentially by the cameras 101A to 101D in accordance witha predetermined frame rate, it is possible to extract a region makingthe difference as a region of the moving object. By storing, as abackground image, an image photographed previously in a state in which aphotographing target region has no moving object and detecting adifference between the photographed image and the background image,alternatively, it is also possible to extract the region making thedifference as the region of the moving object.

The 3D information generating unit 3 generates 3D information of themoving object from the photographed images input from the image inputunit 1 by utilizing the depth information calculated by the depthinformation calculating unit 2. Herein, the 3D information generatingunit 3 generates a set of 3D information for a single photographingtarget region. For example, a set of 3D information is generated fromfour photographed images input to the image input unit 1 from the fourcameras 101A to 101D provided in the first image pickup unit 101. Oneset implies a set of 3D information of moving objects if the movingobjects are extracted from the photographed image.

Specific processing contents of the 3D information generating unit 3will be described below with reference to FIG. 3. FIG. 3 is anexplanatory view showing processing for generating 3D information of amoving object from a plurality of photographed images. As shown in FIG.3, the 3D information generating unit 3 sets a plurality of projectionplanes 31 to 34 in adaptation to a relative angular relationship amongthe cameras 101A to 101D. In the example of FIG. 3, for simpleexplanation, the projection planes 31 to 34 are set in four directionsat an angle of 90 degrees each other.

In other words, the four cameras 101A to 101D are provided in positionsshown in a mark  (for example, wall surfaces in four directions in aroom), and postures of the respective cameras 101A to 101D are set toperform photographing in directions shown in arrows A to D from thosepositions. The 3D information generating unit 3 sets the projectionplanes 31 to 34 in photographing directions corresponding to attachmentpositions and attachment postures of the respective cameras 101A to101D. Information related to the attachment positions and the attachmentpostures of the respective cameras 101A to 101D are registered in the 3Dinformation generating unit 3 by previous calibration.

The 3D information generating unit 3 projects values of respectivepixels of the moving objects included in the photographed images inputby the image input unit 1 (the photographed images of the four cameras101A to 101D in the example of FIG. 3) in accordance with the depthinformation in the directions of the projection planes 31 to 34corresponding to the respective photographed images, thereby generating3D information obtained by synthesizing, into one, the moving objectsincluded in the photographed images.

In other words, the 3D information generating unit 3 draws values ofrespective pixels of the moving objects included in the photographedimage input from the first camera 101A to the image input unit 1 in aposition projected by a distance represented by the depth informationfrom the position of the first camera 101A in the direction of the arrowA in which the projection plane 31 is present. Herein, the value of eachpixel to be drawn may be a value of R, G or B possessed originally bythe photographed image or may be a binary value, a gray scale value orthe like.

The 3D information generating unit 3 also performs the same processingfor the photographed images input from the second to fourth cameras 101Bto 101D to the image input unit 1. In the present embodiment, thus, thesame photographing target region is photographed by the cameras 101A to101D and the values of the respective pixels of the moving objectsincluded in the respective photographed images are synthesized onto asingle image. For this reason, a plurality of pixel values is projectedin overlap onto the same pixel position in a plurality of directions.

In this case, the 3D information generating unit 3 preferentially draws3D information to be synthesized on a front surface of athree-dimensional object as seen from a virtual viewpoint (a point in adirection of a user's side of a display screen) seen by a user. In theexample of FIG. 3, the values of the pixels included in the photographedimages obtained by the first camera 101A and the second camera 101B aredrawn preferentially over the values of the pixels included in thephotographed images obtained by the third camera 101C and the fourthcamera 101D. It is possible to optionally determine which has priority,the photographed image obtained by the first camera 101A or thephotographed image obtained by the second camera 101B. For example, itis possible to perform processing for giving priority to a shorterdistance represented by the depth information.

The overlooking image generating unit 4 synthesizes the 3D informationof the moving objects generated by the 3D information generating unit 3with a space image representing a space of the photographing targetregion, thereby generating an overlooking image of the photographingtarget region. The space image is 3D information representing thephotographing target region three-dimensionally and a space imagegenerated previously is stored in the space image storing unit 8.

The space image stored in the space image storing unit 8three-dimensionally represents a space of an individual photographingtarget region such as each room, a corridor, a step or an elevator(which will be hereinafter referred to as an individual region), andfurthermore, three-dimensionally represents a space of a region having aplurality of photographing target regions compounded (which will behereinafter referred to as a compound region), for example, a wholebuilding or a whole floor.

A user can optionally designate which photographing target region having3D information to be synthesized and which space image for the synthesisby the overlooking image generating unit 4. In other words, the useroperates an operating unit (not shown) to enable designation as to whichindividual region or compound region in the whole building, the wholefloor, a room in a floor, a corridor, a step and an elevator is to bedisplayed as an overlooking image on the display 400.

The operation accepting unit 7 accepts the operation for designating aregion to notify the overlooking image generating unit 4 of thedesignated content. Upon receipt of the notification, the overlookingimage generating unit 4 reads a space image of a designated region(hereinafter referred to as a display target region) from the spaceimage storing unit 8. Then, the 3D information of the moving objectsgenerated by the 3D information generating unit 3 for the display targetregion is synthesized with the read space image to generate anoverlooking image. The operation accepting unit 7 may also notify the 3Dinformation generating unit 3 of the content of the region designationto generate the 3D information of the moving objects for only thedesignated display target region.

The display control unit 5 controls to display, on the display 400, theoverlooking image generated by the overlooking image generating unit 4.As described above, in the present embodiment, the user can designatewhich region has an overlooking image to be displayed on the display 400by the user's operation of the operating unit. The display control unit5 controls to display, on the display 400, the overlooking imagesgenerated by the 3D information generating unit 3 and the overlookingimage generating unit 4 for the designated display target region.

In the present embodiment, moreover, by moving a position of a virtualviewpoint where the user sees the overlooking image (a point in thedirection of the user's side of the display screen) to change theazimuths of the projection planes 31 to 34, it is also possible toperform optional switching into an overlooking image seen at variousangles and to display the overlooking image on the display 400. Theoperation accepting unit 7 accepts an operation for moving the viewpointand notifies the 3D information generating unit 3 and the overlookingimage generating unit 4 of the content of the designation.

The 3D information generating unit 3 sets the projection planes 31 to 34seen from the position of the designated viewpoint and generates the 3Dinformation of the moving objects by the processing described above.Moreover, the overlooking image generating unit 4 deforms the spaceimage read from the space image storing unit 8 into 3D information seenfrom the position of the designated viewpoint, and furthermore,synthesizes the 3D information of the moving objects generated by the 3Dinformation generating unit 3, thereby generating an overlooking image.The display control unit 5 causes the display 400 to display theoverlooking image thus seen from the designated viewpoint.

FIG. 4 is a view showing an example of the overlooking image to bedisplayed on the display 400. FIG. 4(a) shows an overlooking image to bedisplayed when a whole building is designated as a display targetregion. Moreover, FIG. 4(b) shows an overlooking image to be displayedwhen a certain specific floor (herein, a 7^(th) floor) is designated asthe display target region.

In FIG. 4(a), 41 ⁻¹ to 41 ⁻⁹ denote a space image showing each floor inthe building and 42 ⁻¹ and 42 ⁻² denote a space image representing anelevator in the building. A space image of the whole building isconfigured from the space images 41 ⁻¹ to 41 ⁻⁹ of the respective floorsand the space images 42 ⁻¹ and 42 ⁻² of the respective elevators. Thespace image of the whole building does not strictly represent structuresof the respective floors and the elevator and a positional relationshiptherebetween but represent them in simplification.

In FIG. 4(a), moreover, 4 stereoenotes the 3D information of the movingobjects generated by the 3D information generating unit 3. In thepresent embodiment, the photographing target regions are set to eachroom of each floor or the like and the 3D information of the movingobjects are generated by the 3D information generating unit 3 forrespective photographing target regions. When an overlooking image of acompound region such as a whole building is to be displayed as shown inFIG. 4(a), 3D information of the moving objects generated for therespective photographing target regions are synthetized with the spaceimage of the whole building in a corresponding position in each room ofeach floor.

In FIG. 4(b), 44 ⁻¹ to 44 ⁻⁵ denote respective rooms in a floor and 45denotes a corridor in the floor. A space image of the whole floor isconfigured from the respective rooms 44 ⁻¹ to 44 ⁻⁵ and the corridor 45.The space image of the whole floor represents the structures of therespective rooms and the corridor and the positional relationshiptherebetween strictly to some degree.

In FIG. 4(b), furthermore, 46 denotes the 3D information of the movingobjects generated by the 3D information generating unit 3. In an exampleshown in FIG. 4(b), the 3D information of the moving objects generatedby the 3D information generating unit 3 for the respective photographingtarget regions set to each room and a corridor in a floor aresynthesized with a corresponding position in the space image of thewhole floor.

Returning to FIG. 2, description will be given. The relevant informationgenerating unit 6 generates relevant information for relating positionsof each pixel configuring 3D information of moving objects to aphotographed image serving as a projection source of a value of thepixel and stores the relevant information in the relevant informationstoring unit 9. FIG. 5 is a table showing an example of the relevantinformation to be stored in the relevant information storing unit 9.

As shown in FIG. 5, the relevant information has moving object IDs givenuniquely to 3D information of individual moving objects, coordinatepositions on an overlooking image of each pixel configuring the 3Dinformation, and image IDs uniquely given to a photographed imagesserving as a projection source of values of each pixel configuring the3D information.

A single 3D information is generated from a plurality of photographedimages obtained by a plurality of cameras. In other words, there is aplurality of photographed images serving as projection sources of valuesof respective pixels configuring the 3D information. Accordingly, aplurality of image IDs is stored for a single moving object ID.

Herein, the moving object ID and the image ID are given when the 3Dinformation generating unit 3 generates 3D information of a movingobjects from photographed images and the relevant information generatingunit 6 is notified of them. Moreover, coordinate positions on anoverlooking image of each pixel configuring 3D information are specifiedwhen the overlooking image generating unit 4 synthesizes 3D informationwith a space image to generate an overlooking image, and the relevantinformation generating unit 6 is notified of the coordinate position.

The display control unit 5 controls the display 400 to display aphotographed image related to a designated position by referring to therelevant information stored in the relevant information storing unit 9when a position on optional 3D information is designated fromoverlooking images displayed on the display 400 through the operation ofthe operating unit by the user.

In other words, when the operation accepting unit 7 accepts that theoptional position on the overlooking image is designated by theoperation of the user, the display control unit 5 is notified of thedesignated position. The display control unit 5 decides whether or notthe position designated on the overlooking image is a position on the 3Dinformation of the moving object by referring to the relevantinformation stored in the relevant information storing unit 9. If so,the display control unit 5 controls to input a photographed image to bespecified by the image ID related to the designated position from theimage input unit 1 and to cause the display 400 to display thephotographed image.

In the case in which a plurality of image IDs is related to the positionof the 3D information designated on the overlooking image, aphotographed image corresponding to any of the image IDs may bedisplayed selectively on a single screen or photographed imagescorresponding to the image IDs may be divided and displayed on thesingle screen. In the case in which a single photographed image isdisplayed selectively, a selection rule thereof can be set optionally.For example, there is considered a rule for selecting a photographedimage corresponding to the closest projection plane to a position of aviewpoint.

FIG. 6 is a flowchart showing an example of an operation of the imageprocessing device 300 to be one component of the image processing systemaccording to the first embodiment having the structure described above.The flowchart shown in FIG. 6 is started when a power supply of theimage processing device 300 is turned ON.

First of all, the image processing device 300 inputs photographed imagescorresponding to one frame obtained by the image pickup units 101 and102 from the image input devices 201 and 202 (Step S1). Herein, theoperation accepting unit 7 decides whether or not designation of adisplay target region is accepted from a user by the operation of theoperating unit as to which region such as a whole building, a wholefloor, a room in a floor, a corridor, a step or an elevator is to bedisplayed as an overlooking image on the display 400 (Step S2).

If the operation accepting unit 7 accepts the operation for designatinga display target region, the 3D information generating unit 3 and theoverlooking image generating unit 4 change setting to generate 3Dinformation and an overlooking image for the designated display targetregion (Step S3). On the other hand, if the operation accepting unit 7does not accept the operation for designating a display target region,the 3D information generating unit 3 and the overlooking imagegenerating unit 4 do not change the setting described above. In aninitial state, a compound region of the whole building is set to be thedisplay target region, for example.

Next, the operation accepting unit 7 decides whether an operation formoving a position of a virtual viewpoint where the user sees theoverlooking image is accepted or not (Step S4). If the operationaccepting unit 7 accepts the operation of the designated movement, the3D information generating unit 3 and the overlooking image generatingunit 4 set the projection planes 31 to 34 seen in a position of themoved viewpoint (Step S5). On the other hand, if the operation acceptingunit 7 does not accept the operation for moving a viewpoint, the 3Dinformation generating unit 3 and the overlooking image generating unit4 do not change the projection planes 31 to 34 described above.

Then, the depth information calculating unit 2 detects moving objectsfrom each of the photographed images for the display target region inputin the Step S1 (Step S6). Thereafter, the depth information calculatingunit 2 calculates, every pixel, depth information representing adistance from the camera for each of the moving objects detected fromthe respective photographed images (Step S7).

Next, the 3D information generating unit 3 projects a value of each ofpixels of the moving objects detected from the photographed images inthe directions of the projection planes 31 to 34 in accordance with thedepth information every photographing target region included in thedisplay target region. Consequently, the 3D information of the movingobjects are generated every photographing target region (Step S8).

Furthermore, the overlooking image generating unit 4 synthesizes the 3Dinformation of the moving objects generated every photographing targetregion in the Step S8 with a space image corresponding to the displaytarget region (an individual region formed by a single photographingtarget region or a compound region formed by a plurality ofphotographing target regions), thereby generating an overlooking imageof the display target region (Step S9). Then, the display control unit 5causes the display 400 to display the overlooking image thus generated(Step S10).

In a state in which the overlooking image is thus displayed on thedisplay 400, the operation accepting unit 7 decides whether an operationfor the user to designate an optional position on an overlooking imageis accepted or not (Step S11). If the operation accepting unit 7 doesnot accept the position designating operation, the operation acceptingunit 7 further decides whether an operation for turning OFF the powersupply of the image processing device 300 is accepted or not (Step S12).

Herein, if the operation accepting unit 7 does not accept the operationfor turning OFF the power supply, the processing returns to the Step S1and a photographed image of a next frame is input. On the other hand, ifthe operation accepting unit 7 accepts the operation for turning OFF thepower supply, the processing of the flowchart shown in FIG. 6 is ended.

If the operation accepting unit 7 accepts the operation for designatingan optional position on an overlooking image in the Step S11, thedisplay control unit 5 decides whether the position designated on theoverlooking image is a position on the 3D information of the movingobject or not by referring to the relevant information stored in therelevant information storing unit 9 (Step S13).

If the position designated on the overlooking image is not the positionon the 3D information of the moving object, the processing proceeds tothe Step S12. On the other hand, if the position designated on theoverlooking image is the position on the 3D information of the movingobject, the display control unit 5 causes the display 400 to display aphotographed image to be specified by the image ID related to thedesignated position (Step S14).

Then, the operation accepting unit 7 decides whether an operation forreturning the display of the photographed image to that of theoverlooking image is accepted or not (Step S15). If the operationaccepting unit 7 accepts the operation, the processing proceeds to theStep S12. On the other hand, if the operation accepting unit 7 does notaccept the operation, it further decides whether the operation forturning OFF the power supply of the image processing device 300 isaccepted or not (Step S16).

Herein, if the operation accepting unit 7 does not accept the operationfor turning OFF the power supply, the image processing device 300 inputsa next frame of a photographed image corresponding to the image IDspecified in the Step S14 (Step S17). Then, the processing returns tothe Step S14 and the display control unit 5 causes the display 400 todisplay the input photographed image. On the other hand, if theoperation accepting unit 7 accepts the operation for turning OFF thepower supply, the processing of the flowchart shown in FIG. 6 is ended.

As described above in detail, according to the first embodiment, the 3Dinformation of the moving object is generated as data on a group of aplurality of points formed by projection of the values of each pixel inthe directions of the projection planes 31 to 34 in accordance with thedepth information detected from the photographed images and the 3Dinformation are synthesized with the space image of the display targetregion so that the synthesized image is displayed as a single wholeoverlooking image.

For this reason, even if the number of the photographing target regionsis increased in a building having a large scale and a complicated floorstructure or the like, a large number of photographed images are neitherdisplayed on the display 400 by screen division nor displayed byswitching through time division. In other words, a single overlookingimage which has a plurality of photographing target regions in a wholespace of the building, and which is formed by synthesizing the 3Dinformation of the groups of a plurality of points in each of thephotographing target regions is displayed on the display 400.

Consequently, the user can grasp a situation of the whole building at aglance by confirmation of the overlooking image. Moreover, it is alsopossible to eliminate a disadvantage that a display size of anindividual photographed image displayed by screen division is decreasedso that the photographed image is seen with difficulties. According tothe first embodiment, therefore, it is also possible to offer anoverlooking image of a building having a large scale and a complicatedfloor structure so as to be easily understood, and the user can readilygrasp the situation of the whole building.

In general, a polygon model is used for drawing 3D information. Datahaving continuity and high precision is required for changing data onthe group of a plurality of points into a polygon model. It is hard tochange data having high intermittency and low precision into a polygonmodel and it is difficult to unify the data in a three-dimensionalspace. According to the first embodiment, data on the group of aplurality of points having high intermittency and low precision can alsobe used for drawing an image of a moving object which can be identifiedby a man.

According to the first embodiment, moreover, the values of therespective pixels of the moving objects included in the respectivephotographed images obtained by photographing the photographing targetregion at the angles are projected from the photographed images in thedirections of the projection planes of the respective photographedimages so that the 3D information are synthesized. By moving theposition of the virtual viewpoint for seeing the overlooking imageincluding the 3D information of the moving objects to change the azimuthof the projection plane, therefore, it is possible to perform optionalswitching into the overlooking image seen at various angles, therebycausing the display 400 to display the image.

In other words, according to the first embodiment, the overlooking imagein the display target region including the 3D information of the movingobjects processed into the data on the group of a plurality of pointsfrom the photographed images can be displayed on the display 400, andfurthermore, the viewpoint of the overlooking image can be optionallyswitched for the display. By changing the angle of the overlookingimage, consequently, it is also possible to display an object which ishidden behind something and cannot be seen at a certain angle, forexample.

According to the first embodiment, moreover, it is possible to performswitching by optionally selecting which individual or compound regionfrom a whole building, a whole floor, a room in a floor, a corridor, astep and an elevator so as to be the display target region of theoverlooking image. Consequently, it is possible to instantaneouslyperform switching into an overlooking image in an individual region suchas an optional room, a corridor, a step or an elevator as necessary,thereby confirming a motion of moving objects finely whilecomprehensively confirming the motion of the moving objects in eachregion on a single screen through an overlooking image in the compoundregion such as the whole building or the whole floor, for example.

According to the first embodiment, furthermore, each point (each pixel)of the data on the group of a plurality of points configuring the 3Dinformation of the moving object has relevant information for relating aposition on the overlooking image to the photographed image serving asthe projection source of the value of the pixel. By designating theposition of the moving object displayed on the overlooking image,therefore, it is also possible to perform switching into an originalphotographed image in which the designated moving object is present,thereby displaying the photographed image. Consequently, it is alsopossible to monitor the moving object more finely by a photographedimage itself which has not been processed into the data on the group ofa plurality of points.

Second Embodiment

A second embodiment of the present invention will be described belowwith reference to the drawings. FIG. 7 is a diagram showing an exampleof a structure of an image processing system according to the secondembodiment. The second embodiment also describes an example in which theimage processing system according to the present invention is executedin a monitoring system.

As shown in FIG. 7, the monitoring system according to the secondembodiment includes a portable terminal 600 such as a tablet, a notebooktype personal computer or a smartphone in addition to each structureillustrated in FIG. 1. The portable terminal 600 includes a display fordisplaying an image. The portable terminal 600 is connected to an imageprocessing device 300 through an in-house network 500.

FIG. 8 is a block diagram showing an example of a functional structureof the image processing system according to the second embodiment. Sincecomponents having the same symbols as those shown in FIG. 2 have thesame functions in FIG. 8, repetitive description will be omitted.

As shown in FIG. 8, the image processing system according to the secondembodiment includes a position and azimuth detecting unit 11, a displaytarget region specifying unit 12, an image transmitting unit 15 and adisplay control unit 16 in addition to the functional structureillustrated in FIG. 2. Moreover, a 3D information generating unit 13 andan overlooking image generating unit 14 are provided in place of the 3Dinformation generating unit 3 and the overlooking image generating unit4.

Herein, functions of the position and azimuth detecting unit 11 and thedisplay control unit 16 are provided in the portable terminal 600 shownin FIG. 1. Moreover, functions of the display target region specifyingunit 12, the 3D information generating unit 13, the overlooking imagegenerating unit 14 and the image transmitting unit 15 are provided inthe image processing device 300 shown in FIG. 1.

Each of functions of a depth information calculating unit 2, the 3Dinformation generating unit 13, the overlooking image generating unit14, a display control unit 5, a relevant information generating unit 6,an operation accepting unit 7, the display target region specifying unit12 and the image transmitting unit 15 provided in the image processingdevice 300 can also be configured from all of hardware, a DSP andsoftware. If the each function is configured from the software, itactually includes a CPU, an RAM, an ROM and the like in the imageprocessing device 300 and is implemented by an operation of an imageprocessing program stored in a recording medium such as the RAM, theROM, a hard disk or a semiconductor memory.

The position and azimuth detecting unit 11 detects a current position ofthe portable terminal 600 and a current azimuth where the portableterminal 600 is turned. The position and azimuth detecting unit 11 isconfigured from a GPS receiver, for example. The position and azimuthdetecting unit 11 always detects the current position and the currentazimuth of the portable terminal 600 and always transmits the detectedinformation to the image processing device 300 through the in-housenetwork 500.

The display target region specifying unit 12 of the image processingdevice 300 specifies, as a display target region, any of photographingtarget regions which is to be displayed as an overlooking image based onthe current position and the current azimuth of the portable terminal600 which are detected by the position and azimuth detecting unit 11.

FIG. 9 is a view for explaining processing to be performed by thedisplay target region specifying unit 12. Herein, a layout of the floorshown in FIG. 4(b) is illustrated. It is assumed that a monitoringperson holding the portable terminal 600 is present in a corridor 45 ofthe floor and turns the portable terminal 600 in a direction of a room44 ⁻². In this case, a current position PP and a current azimuth PD ofthe portable terminal 600 are detected by the position and azimuthdetecting unit 11 and are transmitted to the image processing device300.

The display target region specifying unit 12 of the image processingdevice 300 specifies, as a display target region for displaying anoverlooking image, a photographing target region of the room 44 ⁻² in adirection of the current azimuth PD from the current position PP of theportable terminal 600 detected by the position and azimuth detectingunit 11.

The 3D information generating unit 13 has the following function inaddition to the function described in the first embodiment. In otherwords, if a display target region is specified by the display targetregion specifying unit 12, the 3D information generating unit 13generates 3D information for a photographing target region included inthe specified display target region by the method described in the firstembodiment. In an example of FIG. 9, only the photographing targetregion of the room 44 ⁻² is included in the display target region.

Herein, it is possible to optionally determine which angle is set toprojection planes 31 to 34 in the generation of the 3D information. Forexample, the angles of the projection planes 31 to 34 are dynamicallyset depending on the current position PP and the current azimuth PD insuch a manner that an overlooking image is displayed as seen in thedirection of the current azimuth PD from the current position PPdetected by the position and azimuth detecting unit 11. Alternatively,the projection planes 31 to 34 may be preset to predetermined angles insuch a manner that the overlooking image is displayed at a fixed angleshown in FIG. 4(b), for example.

The overlooking image generating unit 14 has the following function inaddition to the function described in the first embodiment. In otherwords, in the case in which the display target region is specified bythe display target region specifying unit 12, the overlooking imagegenerating unit 14 generates an overlooking image for the specifieddisplay target region. Then, the overlooking image generating unit 4supplies the generated overlooking image to the image transmitting unit15.

The image transmitting unit 15 transmits the overlooking image generatedby the overlooking image generating unit 14 to the portable terminal600. The display control unit 16 of the portable terminal 600 causes adisplay (not shown) of the portable terminal 600 to display theoverlooking image transmitted by the image transmitting unit 15 as anoverlooking image of the display target region in the direction of thecurrent azimuth PD from the current position PP detected by the positionand azimuth detecting unit 11.

As described above in detail, according to the second embodiment, it ispossible to display a motion of a moving object which is being moved atthe other side of a wall or over another floor as if it is seen throughby using the portable terminal 600. Consequently, when tracking aspecific man, for example, a monitoring person can confirm a motion ofthe man in real time in a different place from a place where the man ispresent. Accordingly, a monitoring capability can be enhanceddramatically.

Although the description has been given to the example in which theoverlooking image is displayed on the display of the portable terminal600 in the second embodiment, the present invention is not restrictedthereto. For example, a portable type display such as an HUD (Head-UpDisplay) may be used. In this case, it is preferable to provide a GPSreceiver or a data transmitting/receiving function on the portable typedisplay.

Third Embodiment

A third embodiment of the present invention will be described below withreference to the drawings. A structure of an image processing systemaccording to the third embodiment is the same as that in FIG. 1 or FIG.6. In other words, the image processing system according to the thirdembodiment is an application example of the first embodiment or thesecond embodiment described above. The third embodiment will bedescribed below as the application example of the first embodiment.

FIG. 10 is a block diagram showing an example of a functional structureof the image processing system according to the third embodiment. Sincecomponents having the same symbols as those shown in FIG. 2 have thesame functions in FIG. 10, repetitive description will be omitted.

As shown in FIG. 10, the image processing system according to the thirdembodiment includes a motion detecting unit 21, a motion pattern storingunit 22, a motion deciding unit 23, an alarm generating unit 24 and amoving object tracking unit 26 in addition to the functional structureshown in FIG. 2. Moreover, a display control unit 25 is provided inplace of the display control unit 5. All of the motion detecting unit21, the motion pattern storing unit 22, the motion deciding unit 23, thealarm generating unit 24, the display control unit 25 and the movingobject tracking unit 26 are provided in the image processing device 300shown in FIG. 1.

Each of functions of a depth information calculating unit 2, a 3Dinformation generating unit 3, an overlooking image generating unit 4,the display control unit 25, a relevant information generating unit 6,an operation accepting unit 7, the motion detecting unit 21, the motiondeciding unit 23, the alarm generating unit 24 and the moving objecttracking unit 26 provided in the image processing device 300 can also beconfigured from all of hardware, a DSP and software. If the eachfunction is configured from the software, it actually includes a CPU, anRAM, an ROM and the like in the image processing device 300 and isimplemented by an operation of an image processing program stored in arecording medium such as the RAM, the ROM, a hard disk or asemiconductor memory.

The motion detecting unit 21 analyzes a change on a time base of 3Dinformation generated by the 3D information generating unit 3, therebydetecting a motion of a moving object represented by the 3D information.In other words, the 3D information is sequentially generated by the 3Dinformation generating unit 3 using a photographed image photographedsequentially by image pickup units 101 and 102 in accordance with apredetermined frame rate. Therefore, the motion detecting unit 21detects an inter-frame difference of the 3D information to be generatedsequentially, thereby detecting the motion of the moving object to berepresented by the 3D information.

In the present embodiment, there is detected a small change in a shapeof the moving object as well as a situation of movement of the movingobject. Since the 3D information is configured from data on a group of aplurality of points, such a small change can also be detected. Forexample, if the moving object is a person, the motion detecting unit 21can also detect a motion of a skeletal structure of a human body.

The motion pattern storing unit 22 prestores data on a specific motionpattern related to the moving object. The motion pattern storing unit 22stores data on a motion pattern, for example, in which a right shoulderis moved upward and a left shoulder is moved downward with suchconsciousness as to lift a heavy baggage when a person holds the baggagein a right hand or a vertical motion is increased more than usual if aperson walks with the heavy baggage held.

The motion deciding unit 2 stereoecides whether or not the motion of themoving object detected by the motion detecting unit 21 is coincidentwith the motion pattern stored in the motion pattern storing unit 22.Herein, the coincidence has a concept including the case in which acoincidence degree is equal to or higher than a predetermined value inaddition to perfect coincidence.

The alarm generating unit 24 generates a predetermined alarm if it isdecided that the motion of the moving object is coincident with themotion pattern by the motion deciding unit 23. For example, the alarmgenerating unit 24 controls the display control unit 25 to display aframe 60 around the moving object decided to be coincident with themotion pattern as shown in FIG. 11. An alarm sound may be made togetherwith the display of the frame 60.

The moving object tracking unit 26 tracks a motion on a time base of themoving object which is decided to be coincident with the motion patternby the motion deciding unit 23. The motion of the skeletal structure ofthe human body to be detected by the motion detecting unit 21 ispeculiar to some degree and can also be utilized for specifyingcontinuity. Consequently, it is possible to specify that 3D informationto be sequentially generated from a photographed image of onephotographing target region are obtained from the same moving objectbased on peculiarity of a motion thereof, thereby tracking the movingobject.

Also in the case in which the moving object is moved across a pluralityof photographing target regions as shown in FIG. 12, moreover, themoving object can be tracked continuously. In other words, it ispossible to easily specify, based on peculiarity of a motion of a movingobject, that 3D information generated for a photographing target regionof a room 44 ⁻² when the moving object is present in the room 44 ⁻² and3D information generated for a photographing target region of a corridor45 after movement of the moving object to the corridor 45 are obtainedfrom the same moving object. Therefore, it is possible to reliably trackthe moving object to be moved across a plurality of photographing targetregions.

The alarm generating unit 24 is notified of a tracking result of themoving object obtained by the moving object tracking unit 26, forexample. Upon receipt of the tracking result, the alarm generating unit24 displays the frame 60 for an alarm through tracking around the movingobject decided to be coincident with the motion pattern by the motiondeciding unit 23.

Although the description has been given to the example in which theanalysis processing is performed by the motion detecting unit 21, themotion deciding unit 23, the alarm generating unit 24 and the movingobject tracking unit 26 in real time together with photographing throughthe image pickup units 101 and 102 in the third embodiment, the presentinvention is not restricted thereto. For example, an overlooking imageto be sequentially generated by the overlooking image generating unit 4may be stored in a database to subsequently perform the analysisprocessing with the stored overlooking image as a target.

By the subsequent analysis processing for the overlooking image storedin the database, moreover, it is also possible to detect acharacteristic motion of a person doing something dishonest in the pastby the motion detecting unit 21 and to store the motion, in the motionpattern storing unit 22, as data on a motion pattern. Thus, it ispossible to easily monitor a suspicious person. In other words, when thesuspicious person enters a photographing target region, the frame 60 canbe displayed around 3D information of the suspicious person to callattention of a monitoring person.

Moreover, it is also possible to extract and display only any of theoverlooking images stored sequentially in the database in a time zoneincluding a moving object having a specific motion. In that case, it isalso possible to display a locus of the moving object and to display aserial motion of the moving object like a strobe picture. Moreover, itis also possible to emphasize and display a portion having a specificchange by the motion of the skeletal structure. Furthermore, it is alsopossible to easily see the motion of the moving object by thinning atime interval to be displayed.

Although the description has been given to the example in which theimage processing system according to the present invention is executedin the monitoring system in the first to third embodiments, the presentinvention is not restricted thereto. For example, it is also possible toexecute the image processing system in a system for analysis of visitorsin a retail store (measurement of the number of people, purchase actionrecognition or the like), safety confirmation or wandering confirmationin a nursing and caring facility or a hospital, or the like.

Although the description has been given to the example in which thestereo camera is used as means for detecting the depth information ofthe moving object in the first to third embodiments, moreover, thepresent invention is not restricted thereto. For example, an ordinarycamera may be used in place of the stereo camera, while a sensor formeasuring a distance using a radar, infrared rays, ultrasonic waves orthe like may be utilized to measure a distance from the image pickupunits 101 and 102 to the moving object.

In order to generate the 3D information of the moving object as pointgroup data, moreover, it is not indispensable to pick up an image byusing a camera. In other words, it is also possible to generate the 3Dinformation of the moving object as point group data having no colorinformation based on distance information (depth information) bymeasuring a distance to the moving object from a device capable ofoutputting the distance information, for example, a radar, an infraredsensor or the like with use of the device. In this case, an appropriatecolor is used to perform drawing.

In addition, all of the first to third embodiments are only illustrativefor concreteness to carryout the present invention and the technicalscope of the present invention should not be thereby construed to berestrictive. In other words, the present invention can be carried out invarious configurations without departing from the gist or main featuresthereof.

1. An image processing system comprising: an image input unit forinputting images photographed respectively by a plurality of camerasprovided to photograph a photographing target region at a plurality ofangles; a depth information calculating unit for calculating, everypixel, depth information representing distances from the cameras ofmoving objects included in a plurality of photographed images input bythe image input unit; a 3D information generating unit for setting aplurality of projection planes in adaptation to a relative angularrelationship between the cameras and projecting values of each of pixelsof the moving objects included in the photographed images input by theimage input unit in directions of the projection planes corresponding tothe respective photographed images in accordance with the depthinformation, thereby generating 3D information obtained by synthesizing,into one, the moving objects included in the photographed images; anoverlooking image generating unit for synthesizing the 3D information ofthe moving object generated by the 3D information generating unit with aspace image representing a space of the photographing target region,thereby generating an overlooking image of the photographing targetregion; and a display control unit for displaying, on a display, theoverlooking image generated by the overlooking image generating unit. 2.The image processing system according to claim 1 further comprising arelevant information generating unit for generating relevant informationrelating positions of respective pixels configuring the 3D informationto a photographed image serving as projection sources of values of thepixels, thereby storing the relevant information in a relevantinformation storing unit, wherein when a position on the 3D informationis designated, the display control unit displays, on the display, aphotographed image related to the designated position by referring tothe relevant information stored in the relevant information storingunit.
 3. The image processing system according to claim 1, wherein thedisplay is configured from a portable type display or a display providedin a portable terminal, and further includes: a position and azimuthdetecting unit for detecting a current position of the portable typedisplay or the portable terminal and a current azimuth where the displayis turned; and a display target region specifying unit for specifying,as a display target region, any of the photographing target regionswhich is to be displayed as the overlooking image based on the currentposition and the current azimuth detected by the position and azimuthdetecting unit, the 3D information generating unit generates the 3Dinformation for the photographing target region included in the displaytarget region specified by the display target region specifying unit,the overlooking image generating unit generates the overlooking imagefor the display target region specified by the display target regionspecifying unit, and the display control unit displays, on the display,the gird's-eye view image of the display target region specified by thedisplay target region specifying unit.
 4. The image processing systemaccording to claim 1 further comprising: a motion detecting unit foranalyzing a change on a time base of the 3D information generated by the3D information generating unit, thereby detecting a motion of the movingobject represented by the 3D information; a motion pattern storing unitfor storing data on a predetermined motion pattern related to the movingobject; a motion deciding unit for deciding whether or not the motion ofthe moving object detected by the motion detecting unit is coincidentwith a motion pattern stored in the motion pattern storing unit; and analarm generating unit for generating an alarm when it is decided thatthe motion of the moving object is coincident with the motion pattern bythe motion deciding unit.
 5. The image processing system according toclaim 4 further comprising a moving object tracking unit for tracking amotion on a time base of the moving object decided to be coincident withthe motion pattern by the motion deciding unit.
 6. An image processingprogram for causing a computer to function as: depth informationcalculating means for analyzing a plurality of photographed images inputrespectively from a plurality of cameras provided to photograph aphotographing target region at a plurality of angles, therebycalculating, every pixel, depth information representing distances fromthe cameras of moving objects included in the photographed images; 3Dinformation generating means for setting a plurality of projectionplanes in adaptation to a relative angular relationship between thecameras and projecting a value of each of pixels of the moving objectsincluded in the photographed images input respectively from the camerasin directions of the projection planes corresponding to the respectivephotographed images in accordance with the depth information, therebygenerating 3D information obtained by synthesizing, into one, the movingobjects included in the photographed images; overlooking imagegenerating means for synthesizing the 3D information of the movingobject generated by the 3D information generating means with a spaceimage representing a space of the photographing target region, therebygenerating an overlooking image of the photographing target region; anddisplay control means for displaying, on a display, the overlookingimage generated by the overlooking image generating means.